Gesture control

ABSTRACT

A method of operating a touchless user interface on an electronic device is disclosed. The electronic device is configured to determine information regarding the position and/or movement of an input object. The method has the steps of: deciding that an engagement gesture has been performed; deciding that a related input gesture has been performed; and carrying out an operation on the device on the basis of the input gesture only if the engagement gesture has been recognized and if the input gesture is one of a subset of possible input gestures determined by the engagement gesture.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to GB Application No. 1412268.3, filedon Jul. 10, 2014 which is incorporated herein by reference in itsentirety.

This invention relates to the control of a device using touchlessgestures, particularly but not exclusively using an ultrasonic touchlesssystem.

In recent years there has been a trend in consumer electronic devices tomove away the use of physical keyboard and button interfaces to more‘natural’ interfaces. These are currently based around touch sensitivescreens and voice recognition, although increasingly interest is growingin the extra possibilities offered by touchless gesture recognition.While this can be used to imitate traditional input methods, it alsoallows intuitive movements to be used to control a device, increasingthe range of control available. WO 2013/132242 gives examples ofpossible control gestures for a device. However, proposed touchlesssystems can still suffer from problems, for example being controlledunintentionally either by background objects or by unintended movementof a hand being interpreted as a control gesture.

The present invention aims to increase the level of control over atouchless device, and when viewed from a first aspect, the inventionprovides a method of operating a touchless user interface on anelectronic device, wherein the electronic device is configured todetermine information regarding the position and/or movement of an inputobject, the method comprising:

deciding that an engagement gesture has been performed;

deciding that a related input gesture has been performed; and

carrying out an operation on the device on the basis of the inputgesture only if the engagement gesture has been recognised and if theinput gesture is one of a subset of possible input gestures determinedby the engagement gesture.

The first aspect of the invention extends to an electronic devicecomprising a touchless user interface and configured to determineinformation regarding the position and/or movement of an input object,the device being further configured to:

decide that an engagement gesture has been performed;

decide that a related input gesture has been performed; and

carry out an operation on the device on the basis of the input gestureonly if the engagement gesture has been recognised and if the inputgesture is one of a subset of possible input gestures determined by theengagement gesture.

Thus it can be seen that an input gesture is not deemed to have beencarried out unless a relevant engagement gesture has already beenperformed and recognised by the device. The engagement gesture forms an‘unlocking’ stage, preparing the device for a user input. This mayreduce the number of accidental inputs by a user, as two input stagesare required in order for an operation to be carried out—the engagementgesture and the input gesture. In addition, the introduction ofengagement gestures associated with specific input gestures may help toreduce the power used by a device, as once an engagement gesture hasbeen recognised, there is a reduced set of possible input gestures,reducing the processing power needed to determine the input gesture.This may, for example, allow part of the touchless interface to beswitched off when a certain engagement gesture is detected.

In an example of such engagement gestures, e.g. to carry out a ‘swipe’movement on a device, in order to change an image being displayed, auser may have to cross at least one of the boundaries of a touchlessgesture zone in order to complete the engagement gesture and prepare thedevice for a swipe gesture. In an alternative example, if a user was tocarry out a circular movement over one corner of the device, crossing inand out of the touchless gesture zone, this may be an engagement gestureto indicate that the user wishes to use that corner to turn a virtualpage of an e-reader application. In this example, touchless sensors inall other corners can be deactivated, as the user has indicated theregion in which the gesture will take place.

In a set of embodiments, the method further comprises providing feedbackto a user that said engagement gesture has been recognised. Suchfeedback may be provided to the user once the engagement gesture hasbeen completed, or may alternatively be provided progressively while theuser is performing the engagement gesture. This may help the userdetermine when the engagement gesture has been recognised and thereforewhen they can perform the input gesture. This in turn aids intuitivelearning of how to use the interface. The feedback may comprise one ormore of: audio, visual, haptic. The feedback may indicate to a user thatthe device is ready to process an input gesture. The feedback may forexample take the form of an audible ‘ping’ when the engagement gestureis complete, a continuous brightening of a screen backlight as theengagement gesture is performed, highlighting of an icon on screen,progressive completion of an icon on screen (e.g. a progress bar),movement of icons on screen that will be required, or vibration of thedevice once the gesture is complete.

In a set of embodiments, the input gesture must be carried out within apredetermined time limit after the engagement gesture has beenperformed. In a set of embodiments, this time limit is between 5 and 20seconds. The introduction of a time limit may help to prevent accidentalinput to the device, as an input gesture performed much later is notprocessed.

In a set of embodiments, every input gesture must be preceded by anengagement gesture. However, in a set of embodiments the performance ofan engagement gesture allows a subset of input gestures to be performedwithout requiring a new engagement gesture. In a set of embodiments,there is a predetermined time limit after the engagement gesture isperformed in which any of the subset of input gestures performed areprocessed. The time limit may for example be between 5 and 30 seconds.Alternatively, the performance of an engagement gesture may allow any ofthe subset of input gestures to be performed, with no new engagementgestures needed. For example, repeating the same or a related inputgesture may not require a new engagement gesture, for example allowing auser to scroll through a plurality of images, either moving the imagesleft or right, without re-performing the engagement gesture. In afurther set of embodiments, a time-out limit is applied after the lastgesture of the subset was performed, after which a time-out limit a newengagement gesture is required.

In the embodiments set out above, the subset of gestures could be asingle gesture.

In a set of embodiments, the engagement gesture comprises a movementwhich is the same as the desired input gesture(s) except that it iscarried out in a different direction. This may help to increase the easewith which a user learns which engagement gesture is required, as it maybe an intuitive extension of the input gesture. In a set of embodiments,the engagement gesture is carried out in the opposite direction to theinput gesture. In an alternative set of embodiments, the engagementgesture is carried out perpendicular to the input gesture.

In a set of embodiments, the subset of desired input gestures maycomprise only one input gesture for each engagement gesture. Thisone-to-one relationship between input gesture and engagement gesture maybe used to greatly reduce the processing power required, as once theengagement gesture has been recognised there is only one possible inputgesture which needs to be processed. However, this may also require auser to learn an engagement gesture for every possible input gesture,increasing the complexity of the system.

While the engagement and input gestures may be distinct and/or differentmotions carried out by the user, this is not essential. In some sets ofembodiments the input gesture is an extension of the engagement gesturein the same direction. It will be appreciated that in such cases themotion of the input object will initially need to be performed such thatit qualifies as an engagement gesture before the remainder of the motionis recognised as an input gesture.

Such arrangements are novel and inventive in their own right and thuswhen viewed from a second aspect, the invention provides a method ofoperating a touchless user interface on an electronic device, whereinthe electronic device is configured to determine information regardingthe position and/or movement of an input object, the method comprising:

deciding that an input gesture has been initiated;

providing feedback to the user;

deciding that the input gesture has been performed beyond a threshold;and

carrying out an operation on the device on the basis of the inputgesture only if the input gesture has been performed beyond thethreshold.

The second aspect of the invention extends to an electronic devicecomprising a touchless user interface and configured to determineinformation regarding the position and/or movement of an input object,the device being further configured to:

decide that an input gesture has been initiated;

provide feedback to the user;

decide that the input gesture has been performed beyond a threshold; and

carry out an operation on the device on the basis of the input gestureonly if the input gesture has been performed beyond the threshold.

Thus it will be seen by those skilled in the art that rather thanrequiring an engagement gesture and distinct input gesture as inaccordance with some embodiments of the previous aspect of theinvention, the user can perform one dynamic and intuitive motion. Thefirst part of the motion may alert the device that a user is performinga gesture and it can then be responsive to the second part of the motionand act accordingly. The feedback advantageously allows the user to seethat the device has detected their gesture and encourages the user tocontinue performing the gesture in order to get the device to carry outthe desired operation.

This may avoid a user's perception of a delay between beginning a motionand when the device begins to respond with an appropriate action.

As previously, the feedback could be discrete or progressive and couldtake any convenient form. In a set of embodiments, the feedbackcomprises a graphical user interface element that provides an indicationthat the device has detected that the input gesture has been initiated.In some such embodiments, the graphical user interface element providesan indication of the input gesture's progress towards the threshold.

In a set of embodiments of either aspect of the invention, the inputgesture is discrete, i.e. the entire input gesture must be completedbefore any action is taken by the device. In an alternative set ofembodiments, the input gesture is continuous, i.e. the device begins toreact before the entire input gesture is completed, for example whenscrolling, albeit after the threshold is reached in the second aspect ofthe invention.

Additionally or alternatively, the subset of desired input gestures maycomprise a plurality of input gestures for each engagement gesture. Theinput gestures may be grouped by the element of the device beingcontrolled, for example using a certain engagement gesture to unlock agraphical user interface element for movement, or a different engagementgesture to activate an application.

The touchless user interface may take any form, for example camera-basedor electromagnetic field-based. However, in a set of embodiments, thetouchless user interface is an acoustic system, preferably an ultrasonicsystem. In a set of embodiments, the system comprises at least one of:transmitters for transmitting signals; receivers for receivingreflections of the transmitted signals; and a processor for determininginformation regarding the position and/or movement of said input objectfrom said reflected signals to determine whether a particular gesturehas been carried out.

In a set of embodiments, the transmitted signals and received signalsare used to calculate channel impulse responses. These may further becombined to form a matrix of channel impulse responses, with the impulseresponse from consecutive time frames adjacent to one another, known asimpulse response images. In a set of embodiments, the impulse responseimages may be analysed to determine whether a gesture has been carriedout.

Certain embodiments of the invention will now be described, by way ofexample only, with reference to the accompanying drawings in which:

FIG. 1 shows a touchless gesture system;

FIGS. 2 a-2 c show a prior art form of gesture control;

FIGS. 3 a-3 e show an exemplary engagement gesture and input gestureembodying the invention;

FIG. 4 shows an engagement gesture in accordance with the invention;

FIG. 5 shows an alternative engagement gesture in accordance with theinvention;

FIG. 6 shows another engagement in accordance with the invention;

FIG. 7 shows another engagement gesture in accordance with theinvention;

FIG. 8 shows a combined engagement and input gesture in accordance withthe invention.

FIG. 1 shows an illustration of the basic principles of a touchlessdetection system. An electronic device such as a smart phone 2 has atouch screen 3 taking up most of the front surface. Around the touchscreen 3 are arranged an ultrasonic transmitter 4 and four ultrasonicreceivers 6, 8, 10, and 12. In this embodiment, the respectivetransducers are provided within the body of the phone 2 butcorresponding apertures are provided in the front glass layer to allowthe air-borne ultrasound to reach them. However, this is not essential,and the transducers may alternatively be on the rear of the device orbelow a complete front glass layer.

A signal generator (not shown) generates signals at ultrasonicfrequencies which are converted to ultrasonic waves by an ultrasonictransmitter 4. These waves bounce off the finger 14 to be tracked, aswell as bouncing off any other obstacles in the vicinity. The reflectedenergy is received by one or more ultrasound receivers 6, 8, 10, 12which convert the energy back into analogue electrical signals which arepassed to a processor (not shown).

The received signals are then processed to calculate the position andmovement of the finger 14. Further exemplary details of how this iscarried out may be found in WO 2009/115799.

The ultrasonic signal might be a spike or a chirp of increasing ordecreasing frequencies. The emitted signal and echoes can be used toestimate the ‘channel impulse response’, that is the response in termsof echoes produced by the environment (‘the channel’) encountered by theultrasound burst. If the emitted signal were a pure impulse, then thereceived signal would be the impulse response. However if the emittedsignal is not such a theoretical pulse (but rather a chirp, forexample), the channel impulse response can be calculated from thereceived signal, using knowledge of the actual emitted signal, to workout what the response would have been had a hypothetical instantaneouspulse been used. Multiple successive impulse responses are preferablyanalysed together by composing them into an impulse response “image”, inwhich consecutive, discretely-sampled impulse responses are alignedside-by-side to form a two-dimensional array (or image if the arrayvalues are represented as greyscale pixels). The impulse responses mayalternatively be combined to form a 3D image.

As described in WO 2009/115799, the impulse response images can then beanalysed to look for patterns which correspond to movements of thefinger by applying a series of filters to the image and determining thestrongest match. When a pattern corresponding to a movement isrecognised with a certain confidence, this can be acknowledged as agesture, and the device can react accordingly. Different patternscorrespond to different gestures, allowing a device to determine whichgesture has been carried out, and to react accordingly.

Of course other techniques could be used instead, e.g. cameras whichproduce a three-dimensional image of the hand.

FIGS. 2 a to 2 c demonstrate a prior art form of gesture control. Inthis example, the hand 20 moves across in front of the screen 22 inorder to change the image displayed on the screen. The hand 20 movesalong a line 24, moving from being off to one side of the screen tobeing above the screen. As the hand 20 moves along this line, the images26 and 28 both move in the same direction at a similar speed. This hasthe effect of the hand appearing to drag the image 28 onto the screen22, and pushing the image 26 off the screen 22. The overall effect isseen in FIG. 2 c, which shows the image 28 filling the screen 22. Thisform of gesture control can be simple and intuitive, but it can lead tounintentional device control, as a user may move their hand over thescreen in this manner without intending to change the on-screen image.It is therefore beneficial to introduce an engagement gesture, such thatthe input gesture is less likely to be confused with ordinary handmovements.

FIGS. 3 a to 3 e demonstrate a touchless gesture control in accordancewith the invention. This involves the use of an engagement gesture inorder to change the image on the screen 32. The user's hand 30 starts atthe edge of the screen 32, on the side from which the image will bechanged (see FIG. 3 a). However, rather than moving directly across thescreen 32, the hand 30 initially moves away from the screen 32. Thistriggers potential recognition of the engagement gesture by the device,which is indicated to the user by a change in the colour of the edge 36of the screen 32 (FIG. 3 b) which brightens as the hand moves away. Oncethe hand 30 has passed a predetermined point 34, the edge 36 reachesfull brightness to indicate that the engagement gesture has beenrecognised and the device is ready to receive a touchless input. Theuser may then move the hand back towards the screen along line 38 (FIG.3 c). Once the hand 30 reaches the edge 36 of the screen 32, scrollingcan begin. The hand can then move across the screen, operating a controlgesture as seen in FIGS. 2 a to 2 c and pushing image 40 off the screen32 while pulling image 42 on to the screen 32 (FIG. 3 d). Once the hand30 has moved a set distance across the screen 32, the image 42 is pulledfully onto the screen 32 (FIG. 3 e).

FIGS. 4 to 8 demonstrate alternative engagement gestures.

FIG. 4 demonstrates an engagement gesture which replicates the inputgesture, but is carried out partly in a direction perpendicular to thatof the input gesture. The engagement gesture is demonstrated with blackarrows, and the input gesture is demonstrated with white arrows, whichindicate the movement of a hand. The hand initially moves in onedirection across the screen 44, as indicated by arrow 46. Once the handhas reached an upper edge of the screen 44, it can be moved in anymanner to the right hand side of the screen 44, in preparation for theinput gesture. The hand must move past a predetermined distance 48,before moving towards the screen 44 along arrow 50. This completes theengagement gesture, and any further movement across the screen 44 alongarrow 52 is taken as the input gesture, and causes the display of thescreen 44 to change.

FIG. 5 demonstrates an alternative engagement gesture. In this example,the engagement gesture requires a hand to move away from the screen 54along a path 56 until it passes a predetermined point 58. The hand mustthen return towards the screen 54 along a path 60. At this point theengagement gesture is complete, and the device is ready to determine aninput gesture comprising the hand moving across the screen 54 along thearrow 62, carrying out an input gesture.

The engagement gesture shown in FIG. 6 is a version of that in FIG. 5with increased complexity. In this example, there are two limitingdistances 64 and 66, both of which are on the same side of the screen68. These provide the limits within which the engagement gesture must becarried out. The engagement gesture requires the hand to follow thethree arrows 70, 72, 74, which cross the inner limit 64, then the outerlimit 66, before crossing the inner limit 64 for a second time. Once theinner limit 64 has been crossed a second time, the engagement gesturehas been completed, and the user can carry out the swipe motion 76,changing the display on the screen 68.

Another example of an engagement gesture is shown in FIG. 7. The handmust cross the screen 78 twice, both in directions perpendicular to thatof the input gesture, in order to complete the engagement gesture. Therequired movements are demonstrated by arrows 80 and 82. The hand mustcross from the bottom to the top of the screen 78, before returning tothe bottom. Any movement which is not directly over the screen is notrelevant to the gesture, and the hand can move in any manner at thesetimes, provided it continues the engagement gesture or begins the inputgesture within a specified time frame. Once the engagement gesture iscompleted, the swipe movement 84 can be carried out across the screen,causing a change in the display.

FIG. 8 shows a combined engagement and input gesture in accordance withaspects of the invention. The hand 90 is moved from right to left acrossthe touch screen 3 as indicated by the arrow 92. The motion is detectedby the ultrasonic receivers 6, 8, 10, and 12 as outlined previously.Once the device has determined that the user is attempting to perform aswipe gesture, a GUI arrow 94 is displayed on the touch screen 3 andbegins to “fill up” from right to left in real-time in order to indicateto the user that the device 2 has recognised that the user may beinitiating an input. The GUI arrow 94 indicates that they must continueperforming the right to left motion in order to completely “fill” thearrow before the device 2 will change the display.

While each of these exemplary gestures are demonstrated on a movementfrom right to left across a screen, they are equally applicable formovements in all directions across a screen, by being rotatedaccordingly, and can also be applied to non-swipe gestures across ascreen. These gestures may for example be a pinch, a circle, anapproach/retreat from the screen or a twist. These gestures may forexample cause on-screen objects to change size, shape or colour, or maycause different functionality within an application, or cause differentapplications to open or close depending on the input gesture which isdetermined WO 2013/132242 gives examples of different touchless gestureswhich can be used for input, and different situations in which theycould be used. However, these are simply examples of the possiblegestures, and are not limiting to this application.

1. A method of operating a touchless user interface on an electronicdevice, wherein the electronic device is configured to determineinformation regarding the position and/or movement of an input object,the method comprising: deciding that an engagement gesture has beenperformed; deciding that a related input gesture has been performed; andcarrying out an operation on the device on the basis of the inputgesture only if the engagement gesture has been recognised and if theinput gesture is one of a subset of possible input gestures determinedby the engagement gesture.
 2. The method as claimed in claim 1 furthercomprising providing feedback to a user that said engagement gesture hasbeen recognised.
 3. The method as claimed in claim 2 wherein thefeedback is provided to the user once the engagement gesture has beencompleted.
 4. The method as claimed in claim 2 wherein the feedback isbe provided progressively while the user is performing the engagementgesture.
 5. The method as claimed in claim 2 wherein the feedbackcomprises visual feedback.
 6. The method as claimed in claim 2 whereinthe feedback comprises audio feedback.
 7. The method as claimed in claim2 wherein the feedback comprises haptic feedback.
 8. The method asclaimed in claim 2 wherein the feedback indicates to the user that thedevice is ready to process an input gesture.
 9. The method as claimed inclaim 4 wherein the feedback comprises continuous brightening of ascreen backlight as the engagement gesture is performed.
 10. The methodas claimed in claim 2 wherein the feedback comprises highlighting anicon on a screen of the device.
 11. The method as claimed in claim 4wherein the feedback comprises progressive completion of an icon on ascreen of the device.
 12. The method as claimed in claim 2 wherein thefeedback comprises a movement of at least one icon on a screen of thedevice.
 13. The method as claimed in claim 2 wherein the feedbackcomprises a vibration of the device once the gesture is complete. 14.The method as claimed in claim 1 wherein the input gesture must becarried out within a predetermined time limit after the engagementgesture has been performed.
 15. The method as claimed in claim 14wherein the time limit is between 5 and 20 seconds.
 16. The method asclaimed in claim 1 wherein every input gesture must be preceded by anengagement gesture.
 17. The method as claimed in claim 1 wherein theperformance of an engagement gesture allows a subset of input gesturesto be performed.
 18. The method as claimed in claim 17 wherein any ofthe subset of input gestures performed are processed within apredetermined time limit after the engagement gesture is performed. 19.The method as claimed in claim 17 wherein a time-out limit is appliedafter a last gesture of the subset was performed, and after saidtime-out limit a new engagement gesture is required.
 20. The method asclaimed in claim 17 wherein the subset of gestures is a single gesture.21. The method as claimed in claim 1 wherein the engagement gesturecomprises a movement which is the same as the desired input gesture(s)except that it is carried out in a different direction.
 22. The methodas claimed in claim 21 wherein the engagement gesture is carried out inthe opposite direction to the input gesture.
 23. The method as claimedin claim 21 wherein the engagement gesture is carried out perpendicularto the input gesture.
 24. The method as claimed in claim 1 wherein thesubset of desired input gestures may comprise only one input gesture foreach engagement gesture.
 25. The method as claimed in claim 1 whereinthe input gesture is an extension of the engagement gesture in the samedirection.
 26. The method as claimed in claim 1 wherein the inputgesture is discrete.
 27. The method as claimed in claim 1 wherein theinput gesture is continuous.
 28. The method as claimed in claim 1wherein the subset of desired input gestures may comprise a plurality ofinput gestures for each engagement gesture.
 29. The method as claimed inclaim 1 wherein the touchless user interface is an acoustic system. 30.The method as claimed in claim 29 wherein the acoustic system is anultrasonic system.
 31. The method as claimed in claim 29 wherein theacoustic system comprises at least one of: transmitters for transmittingsignals; receivers for receiving reflections of the transmitted signals;and a processor for determining information regarding the positionand/or movement of said input object from said reflected signals todetermine whether a particular gesture has been carried out.
 32. Themethod as claimed in claim 31 wherein the transmitted signals andreceived signals are used to calculate channel impulse responses. 33.The method as claimed in claim 32 wherein the calculated channel impulseresponses are combined into a matrix of channel impulse responses. 34.The method as claimed in 33 wherein the matrix of channel impulseresponses is analysed to determine whether a gesture has been carriedout.
 35. An electronic device comprising a touchless user interface andconfigured to determine information regarding the position and/ormovement of an input object, the device being further configured to:decide that an engagement gesture has been performed; decide that arelated input gesture has been performed; and carry out an operation onthe device on the basis of the input gesture only if the engagementgesture has been recognised and if the input gesture is one of a subsetof possible input gestures determined by the engagement gesture.
 36. Amethod of operating a touchless user interface on an electronic device,wherein the electronic device is configured to determine informationregarding the position and/or movement of an input object, the methodcomprising: deciding that an input gesture has been initiated; providingfeedback to the user; deciding that the input gesture has been performedbeyond a threshold; and carrying out an operation on the device on thebasis of the input gesture only if the input gesture has been performedbeyond the threshold.
 37. The method as claimed in claim 36 wherein thefeedback comprises a graphical user interface element that provides anindication that the device has detected that the input gesture has beeninitiated.
 38. The method as claimed in claim 37 wherein the graphicaluser interface element provides an indication of the input gesture'sprogress towards the threshold.
 39. The method as claimed in claim 36wherein the input gesture is discrete.
 40. The method as claimed inclaim 36 wherein the input gesture is continuous.
 41. The method asclaimed in claim 36 wherein the touchless user interface is an acousticsystem.
 42. The method as claimed in claim 41 wherein the acousticsystem is an ultrasonic system.
 43. The method as claimed in claim 41wherein the acoustic system comprises at least one of: transmitters fortransmitting signals; receivers for receiving reflections of thetransmitted signals; and a processor for determining informationregarding the position and/or movement of said input object from saidreflected signals to determine whether a particular gesture has beencarried out.
 44. The method as claimed in claim 43 wherein thetransmitted signals and received signals are used to calculate channelimpulse responses.
 45. The method as claimed in claim 44 wherein thecalculated channel impulse responses are combined into a matrix ofchannel impulse responses.
 46. The method as claimed in 45 wherein thematrix of channel impulse responses is analysed to determine whether agesture has been carried out.
 47. An electronic device comprising atouchless user interface and configured to determine informationregarding the position and/or movement of an input object, the devicebeing further configured to: decide that an input gesture has beeninitiated; provide feedback to the user; decide that the input gesturehas been performed beyond a threshold; and carry out an operation on thedevice on the basis of the input gesture only if the input gesture hasbeen performed beyond the threshold.